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EMAIL MESSAGING PROGRAM 
WITH BUILT-IN VIDEO AND/OR 
AUDIO MEDIA RECORDING 
AND/OR PLAYBACK 
CAPABILITIES 

Background of Invention 

[0001] This invention relates generally to messaging programs, such as email messaging 
programs, and more particularly to such programs that have built-in video and/or 
audio media recording and/or playback capabilities. 

[0002] The popularity of small video cameras for use with personal computers has been 
increasing. Such video cameras are sometimes referred to as netcams or webcams. 
Typically, they connect to the Universal Serial Bus (USB) or other port of a computer, 
and enable users to record or stream video and audio into the computer. Popular 
applications of such cameras include small-scale video conferencing, live transfer of 
video onto a web site, as well as other applications. 

[0003] Unfortunately, using such cameras in connection with existing email messaging 
programs is generally not possible. An email messaging program is defined herein as 
a program that can both send and receive email. Such a program is typically, but not 
necessarily, able to accommodate email according to a given standard, such as the 
Post Office Protocol (POP), or the Internet Messaging Access Protocol (IMAP). Such 
programs include versions of Microsoft Outlook, available from Microsoft Corp., of 
Redmond, Wash., as well as versions of America Online (AOL), available from AOL 
Time Warner, of New York, N.Y. 
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[0004] A user wishing to send video or audio media with such an existing email 

messaging program has only limited options. The user can record the video or audio 
media with another program, completely separate from the email messaging program, 
and then attach the recorded media as a file attachment to an email message within 
the email messaging program. This approach, however, is disadvantageous. It 
requires the user to go outside the email messaging program to perform video or 
audio media recording capability. Furthermore, especially in the case of video, the 
resulting recorded media may have a large file size. However, many Internet Service 
Providers (ISP's) restrict the size of email messages. 

p [0005] Even if the user is able to send the recorded video or audio with his or her ISP, the 

at. 

f 11 recipient's ISP may still reject the email message as too large. Besides individual email 
message size limits, another constraint imposed by many ISP's and email providers, 

W 

\f) especially web-based email providers such as Hotmail, run by Microsoft Corp., is a 

J'l limit on total email mailbox size. An email provider may thus also reject an email 

3 message that has recorded audio or video because receiving it would cause the 

$** 

Sj recipient to exceed his or her allowed total email mailbox size. 

raja 

O [0006] The user may also record video or audio media and upload the resulting file to an 
flj Internet web site, such as by using the File Transfer Protocol (FTP). In this case, the 

user provides the recipient of the email message with instructions as to how to 
download the media. Certain web sites also host such video or audio media for this 
purpose. However, in either situation, the user still must inconveniently perform some 
manual steps in order to upload the media to the web site. The recipient is similarly 
inconvenienced, having to navigate a perhaps unfamiliar web site in order to retrieve 
the media. 

[0007] For these and other reasons, therefore, there is a need for the present invention. 

Summary of Invention 

[0008] 

The invention relates to an email messaging program with built in video and/or 
audio media recording and/or playback capabilities. A system includes a first client 
and a second client, each having an email messaging program installed thereon. The 
user of the first client composes a message and records media. In a streaming media 
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embodiment, the email messaging program of the first client uploads the media to a 
streaming media server, and sends the message to the user of the second client. In a 
file attachment embodiment, the email messaging program of the first client attaches 
the recorded media to the message, and sends the message to the user of the second 
client. 

[0009] The email messaging program of the second client retrieves the message. In the 
streaming media embodiment, when the user of the second client views the message, 
the email messaging program of the second client downloads the media from the 
streaming media server, and plays back the media for the user of the second client. In 
O the file attachment embodiment, when the user of the second client views the 

ffl message, the email messaging program plays back the media as attached to the 



message for the user of the second client. 



[0010] Embodiments of the invention provide for advantages over the prior art. The email 
messaging program of the invention has built-in video and/or audio media recording 
and/or playback capabilities. As a result, a user of the email messaging program does 
not have to exit the program, and access a separate program, in order to take 



1 1 

III 



P advantage of such capabilities. Furthermore, in the streaming media embodiment, the 

o 

ry recorded media does not actually reside within or as an attachment to an email 

message. Therefore, email messages sent by and received with an email messaging 
program according to such an embodiment will not typically exceed individual email 
size and total email mailbox size limitations. 



[0011] 



Having built-in audio and/or video recording in particular is advantageous for 
other reasons as well. A user can send email with recorded media using his or her 
everyday email messaging program, without having to resort to another program. 
Thus, the user can use the address book typically found in such a program to select 
recipients of the email, and can easily forward received messages that have video. The 
user can use the auto-signature option typically found in such a program to 
automatically append signatures to sent and forwarded messages with video. Such 
features are not available with separate email messaging and media programs. The 
user may also use audio and/or video when replying to a received email message, 
which is particularly useful in smaller-sized devices, such as personal digital assistant 
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(PDA) devices and cellular phones, that lack full-sized keyboards on which to enter a 
text reply. Furthermore, having built-in audio and/or video playback allows the user 
to easily replay recorded messages that were received and that contain video. 

[0012] Still other aspects, advantages, and embodiments of the invention will become 
apparent by reading the detailed description that follows and by referring to the 
accompanying drawings. 

Brief Description of Drawings 

[0013] FIG. 1 is a diagram of a system according to a streaming media embodiment of 
the invention. 

[0014] FIG. 2A is a flowchart of a method for sending an email message having 

associated streaming media, according to a streaming media embodiment of the 
invention. 

[001 5] FIG. 2B is a flowchart of a method for receiving an email message having 

associated streaming media, according to a streaming media embodiment of the 
invention. 

[001 6] FIG. 3 is a diagram of a system according to a file attachment embodiment of the 
invention. 

[001 7] FIG. 4A is a flowchart of a method for sending an email message having media 
attached thereto, according to a file attachment embodiment of the invention. 

[001 8] FIG. 4B is a flowchart of a method for receiving an email message having media 
attached thereto, according to a file attachment embodiment of the invention. 

[001 9] FIG. 5 is a diagram of a client device, according to an embodiment of the 
invention. 

[0020] FIG. 6 is a diagram of an email messaging program, according to an embodiment 
of the invention. 

[0021] 

FIG. 7 is a diagram showing an example screenshot of how an email messaging 
program with built-in audio and/or video recording and/or playback capabilities may 
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appear to the user, according to an example embodiment of the invention. 

Detailed Description 

[0022] In the following detailed description of exemplary embodiments of the invention, 
reference is made to the accompanying drawings that form a part hereof, and in which 
is shown by way of illustration specific exemplary embodiments in which the invention 
may be practiced. These embodiments are described in sufficient detail to enable 
those skilled in the art to practice the invention. Other embodiments may be utilized, 
and logical, mechanical, and other changes may be made without departing from the 
spirit or scope of the present invention. The following detailed description is, 
therefore, not to be taken in a limiting sense, and the scope of the present invention 



CP is defined only by the appended claims. 

11% 

Ed 



[0023] Streaming Media Embodiment 



Ifl [0024] FIG. 1 shows a system 100 according to a streaming media embodiment of the 



invention. The system 1 00 includes a first client 1 02 and a second client 1 04. Each of 
W the clients 1 02 and 1 04 is generally a platform on which programs can be run, and is 

distinguished from the term client as can alternatively be used to describe a computer 
program such as an email messaging program. The clients 102 and 104 are 
communicatively coupled to a network 1 06. The network 1 06 may be one or more of 
the Internet, an intranet, an extranet, a local-area network (LAN), a wide-area network 
(WAN), a telephony network like the public switched telephone network (PSTN), the 
Integrated Services Digital Network (ISDN), cellular or other types of wireless 
telephony networks, and so on. 

[0025] The first client 1 02 has running thereon an email messaging program 1 1 0. The 
email messaging program 1 1 0 allows a user of the first client 1 02 to perform 
functionality related to email messages. Such functionality includes composing and 
sending emails, receiving emails, replying to emails, forwarding emails, and so on. 
The email messaging program 100 also has built-in audio and/or video playback 
and/or recording capabilities. The phrase audio and/or video is inclusive of both 
audio and video, as well as audio only and video only. More generally, the phrase 
audio and/or video is referred to as media, such as multimedia. 
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[0026] The term recording as used herein is used in a general and all-encompassing 

sense. The term can mean storing audio and/or video for playback at a later time. It 
can also mean the capture, and/or encoding (such as compressing), and/or 
transmitting of such media. Thus, the media might be played back immediately, or 
played back later. That is, the persistent storage of the media does not have to occur 
to fall within the term recording as used herein. Similarly, the term playing back can 
include the playing back of previously stored media, as well as playing back live 
media. 

On the first client 1 02, the user uses the email messaging program 11 0 to 
compose an email message 1 12, as well as to record media 1 14 that is associated 
with the email message 1 1 2, as indicated by the dotted line 1 1 6. The email message 
1 12 typically includes text, and may only include text. Alternatively, the email 
message 1 12 may include images as well as text, formatted in accordance with a 
markup language, such as the HyperText Markup Language (HTML). As the media 1 14 
is recorded, or after the media 1 1 4 is recorded, it is streamed, or uploaded, to a 
streaming media server 1 08 also communicatively coupled to the network 1 06. This is 
indicated by the line 1 1 8. The media 1 1 4 is stored at the streaming media server 1 08 
as the media 1 14\ 

The email message 1 12 is sent by the email messaging program 1 10 through the 
network 1 06 for ultimate receipt by the email messaging program 1 22 of the second 
client 1 04, as the email message 1 1 2'. This is indicated by the line 1 20. The email 
messaging programs 1 10 and 122 may be compatible with email standards such as 
the Post Office Protocol (POP), the Internet Messaging Access Protocol (IMAP), and/or 
other standards, or they may be proprietary in nature. Typically, the email message 
1 12 is received by an email server (not shown in FIG. 1), from which the email 
messaging program 1 22 downloads the email message 1 1 2, as the email message 
1 12\ Alternatively, the email message can be accessed over the Internet, using a web- 
based email messaging program. 

[0029] 

The user of the second client 1 04 may at some point open the email message 1 1 2' 
using the email messaging program 1 22, or otherwise request that the email 
messaging program 1 22 display the email message 1 1 2' for viewing. At this time, the 
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email messaging program 122 automatically or upon the user's command downloads 
or streams the media 1 14' from the streaming media server 108, over the network 
106, as the media 1 14". This is because the media 1 14" is associated with the email 
message 1 1 2\ as indicated by the dotted line 1 1 6'. The email messaging program 1 22 
may first download the entirety of the media 1 14" before playing it back to the user of 
the second client 1 04, as indicated by the line 1 24, or may play the media 1 1 4" as it is 
downloaded, which is known as streaming. 

[0030] The system 1 00 has been described as the email messaging program 1 1 0 of the 
first client 1 02 sending a composed email message 1 1 2, and the email messaging 

C3 program 122 of the second client 104 receiving the email message 1 12'. However, 

•«.*! 

both email messaging programs 1 1 0 and 1 22 typically have both sending and 

r* receiving capabilities. That is, the email messaging program 122 of the second client 

Ul 

%[l 1 04 may also allow the user of the second client 1 04 to compose an email message 

and record associated media, and the email messaging program 1 1 0 of the first client 
9 ■ 1 02 may also allow the user of the first client 1 02 to receive an email message and 

m play back associated media. 

^« 

CIS [0031] FIG. 2A shows a method 200 that outlines the basic process followed by an email 
fjj messaging program according to a streaming embodiment of the invention to send an 

email message having associated media. First, a message entered by the user of the 
client on which the email messaging program runs is saved by the email messaging 
program (202). The term saving is used herein in a general sense. The email 
messaging program may save the message to persistent storage, such as a hard disk 
drive, or keep it in volatile memory. Concurrently, after, or before the entering and 
saving of this email message, media is recorded (204). As previously indicated, the 
media may include audio, video, or both audio and video. As the media is recorded, or 
after it has been entirely recorded, the media is uploaded to a streaming media server 
over a network (206). The email message is sent over the network to its recipient, with 
a link or other entity indicating that the message has associated media that has been 
uploaded to the server (208). It is noted that the steps and/or acts indicated as the 
method 200 may be performed in orders other than that indicated in FIG. 2A, as is the 
case for other methods of other embodiments of the invention. For instance, the email 
message may initially be sent, and then the recorded media may be uploaded. 
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[0032] FIG. 2B shows a method 250 that outlines the basic process followed by an email 
messaging program according to a streaming embodiment of the invention to receive 
an email having associated media. A message is received over a network (252). In 
response to a user requesting the email messaging program to display the message, 
the email message is displayed (254). This request can be on a per-message basis, 
such as by the user selecting the particular email message, or on a default basis. For 
instance, the email messaging program may have a preview pane in which messages 
are automatically displayed. The email messaging program downloads the associated 
media from the streaming media server that has been previously uploaded to the 
server (256), and plays back this media (258). Playback may be performed as the 

if) 

media is being downloaded from the server, or after the media has been downloaded 

V* from the server in its entirety. 

CO 

W 

[0033] Playback can be performed in a number of different manners, and is not limited to 



If! 



y 
u 

M 
n. 



integration with the email messaging program, although having the email messaging 
program have code for media playback is within the confines of the invention. For 
instance, a web-based email messaging program can be used, which is not installed 
on the user's computer. As another example, the user may be using an installed email 
messaging program, but the code that actually plays back the media may not be part 
of the program itself. The playback may be accomplished by a subsidiary program, 
such as a Java applet that has tags embedded in an HTML email message, and the 
code of which is downloaded in real-time and executed when the email messaging 
program renders the HTML message. Other such players can also be embedded into 
the HTML message, such as the Apple QuickTime player, available from Apple 
Computer of Cupertino, Calif., and the Microsoft Windows Media Player, available from 
Microsoft Corp. Furthermore, in some instances, an HTML email message or a text 
email message may contain a Universal Resource Locator (URL) link therein that the 
user must explicitly select to launch a web browser window to display a page 
including a media player. 

[0034] File Attachment Embodiment 

[0035] 

FIG. 3 shows a system 300 according to a file attachment embodiment of the 
invention. The system 300 includes the first client 1 02 and the second client 1 04, as 
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before, and both the clients 102 and 104 are communicatively coupled to the network 
1 06, also as before. The first client 1 02 has running thereon the email messaging 
program 1 10. The email messaging program 1 10 allows the user of the first client 
102 to perform functionality related to email messages, and also has built-in audio 
and/or video playback and/or recording capabilities. 

[0036] On the first client 102, the users uses the email messaging program 1 10 to 

compose an email message 302, as well as to record media 304 that is associated 
with the email message 302. After the media 304 has been recorded, it is saved as a 
file and attached as a file attachment to the email message 302, as indicated by the 
line 306. The email message 302, with the media 304 as a file attachment, is sent by 
the email messaging program 1 1 0 through the network 1 06 for ultimate receipt by 
the email messaging program 122 of the second client 104, as indicated by the line 
308. The email message 302 having the media 304 attached as a file attachment is 
received at the second client 1 04 as the email message 302' having the media 304 f as 
a file attachment, the latter indicated by the line 306\ 

[0037] When the user of the second client 1 04 opens the email message 302', or 

otherwise requests that the email messaging program 122 display the email message 
302* for viewing, the email messaging program 1 22 plays back the media 304' 
associated with the email message 302'. The email messaging program 122 may 
automatically recognize that the file attachment to the email message 302' is a media 
file, such as audio and/or video, and playback the media. Thus, the email messaging 
program 122 also has built-in audio and/or video capabilities. Furthermore, playback 
may be accomplished in alternative manners, as has been described. In the case of a 
file attachment, opening the attachment in the email messaging program may launch 
a separate viewing program associated with the file attachment for playing back the 
media. 



As before, whereas the system 300 has been described as the email messaging 
program 1 10 of the first client 102 performing the media recording and message 
sending, and the email messaging program 122 of the second client performing the 
media playback and message receiving, either program 1 10 or 122 may perform 
either of these functionalities. The primary difference between the system 100 of FIG. 



[0038] 



APP ID=09683995 



Page 9 



UJ 



1 and the system 300 of FIG. 3 is that the email messages sent and received within 
the system 1 00 do not include media, but rather links to media on a streaming media 
server, whereas the email messages sent and received within the system 300 include 
media as file attachments. As such, the system 100 uses an intermediary streaming 
media server to at least temporarily store the media, whereas the system 300 does not 
use an intermediary streaming media server. 

[0039] FIG. 4A shows a method 400 that outlines the basic process followed by an email 
messaging program according to a file attachment embodiment of the invention to 
send an email message having associated media. First, a message entered by the user 
of the client on which the email messaging program runs is saved by the email 
messaging program (402), where the term save is used in a general sense, as has 
fjj been described. Concurrently, after, or before the entering and saving of this email 

message, media is recorded (404). Once the media has been recorded, it is attached 

as a file attachment to the email message (406). The email message is finally sent over 

If! 

a network to its recipient (408). 

if*! 

UJ [0040] FIG. 4B shows a method 450 that outlines the basic process followed by an email 
messaging program according to a file attachment embodiment of the invention to 
receive an email having associated media. First, a message is received over a network 
(452), where the message has media attached thereto as a file attachment. In 
response to a user requesting the email messaging program to display the message, 
the email message is displayed (454). The media that is attached to the email 
message as a file attachment is then played back (456). 

[0041] Client with Email Program Having Built-in Media Recording and/or Playback 
Capability 

[0042] p| C 5 s hows an example client device 500 according to an embodiment of the 
invention. The client device 500 may implement one or both of the clients 102 and 
1 04 of FIG. 1 . The client device 500 can be a desktop or a laptop computer, as well as 
other types of computerized devices. Such computerized devices may include personal 
digital assistant (PDA) devices, cellular and other types of wireless phones, add-on 
devices to displays known as set-top boxes, and so on. The client device 500 typically 
but not necessarily includes an operating system (OS) 502, an email messaging 



APP K>=09683995 



Page 10 of 25 



program 504, a network mechanism 506, and audio and/or video hardware 508. For 
instance, cellular phones and other types of wireless phones in particular may not 
have an integrated OS. The client device 500 may also include other software, 
hardware, mechanisms, components, and so on. 

[0043] When it is present, the OS 502 is the master control program that runs the 

computer. It is usually the first program loaded when the computer is turned on, and 
its main part, the "kernel," resides in memory at all times. The OS 502 typically sets 
the standards for all programs that run on the computer. The email messaging 
programs 504 may communicate with the OS 502 for user interface and file 

fftl 

k *J management operations. Examples of the OS 502 include versions of the Microsoft 

\Q 

CO Windows OS, available from Microsoft Corp., versions of the Apple Macintosh OS, 

CO 

w 



p a 



available from Apple Computer, Inc., of Cupertino, Calif., versions of the Linux OS, 
versions of the UNIX OS, and so on. 



[0044] The email messaging program 504 allows a user to perform email message- 
W related functionality, as has been described. Furthermore, the email messaging 

program 504 includes built-in audio and/or video recording and/or playback 
capability, as has also been described. The email messaging program 504 runs or in 



S3 

lil conjunction with the OS 502. The networking mechanism 506 allows the client device 



[0045] 



500 to communicatively couple to a network. The mechanism 506 may be one or 
more of an analog modem, an ISDN adapter, a network adapter card, such an Ethernet 
card, a network adapter chipset, and so on. The mechanism 506 may also be one or 
more of a cable modem, a Digital Subscriber Loop (DSL) modem, a digital modem, and 
a wireless modem. The audio and/or video hardware 508 can include both recording 
and playback hardware. Such hardware may include speakers, microphones, display 
devices, and video cameras. The video camera may be a web cam, a net cam, a digital 
video (DV) cam with a FireWire (IEEE— 1 394) or other interface, a Hi-8 cam with an S- 
video or other type of interface, and so on. 

FIG. 6 shows the email messaging program 504 in more detail, according to an 
embodiment of the invention. The program 504 includes composing and recording 
capability 602, and optionally may include built-in viewing and playing back capability 
604. Each of these capabilities may be implemented as a software and/or hardware 
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mechanism, component, module, and so on, and each may be considered the means 
to perform its respective functionality. Other components and capabilities may be 
included in the program 504. Furthermore, the capabilities 602 and 604 may be 
programmed in a different manner than that shown in FIG. 6. For instance, the 
message composing and viewing capability may be programmed as a component 
separate from the component as which the media recording and playback capability is 
programmed. 

[0046] The composing and recording capability 602 allows a user to compose a message 
and record media associated with the message, to send to a user over a network via a 

P networking mechanism. The media may be associated with the message as streaming 

- n 

ffl media, a media attached as a file attachment to the message, or in another manner. 

f ^ The viewing and playback capability 604 allows a user to view a message received 

W 

vp over a network via a networking mechanism, and play back media associated with the 



I?! 



message. 



P [0047] The email messaging program 504 can be programmed in a variety of different 
P computer programming languages. In one embodiment, the program 504 is 

Q programmed in a traditional programming language such as C or C++, or a language 

?*% 

fy such as Java or C#. The media playback and recording capabilities may be 

implemented in one embodiment by embedding one or more media controls 
consistent with the mark-up language, such as an ActiveX control or Java applet in the 
case of HTML, as part of the program 504, or as part of the email message itself. 

[0048] FIG. 7 shows an example screenshot of how an email messaging program with 
built-in media recording capability may appear to the user, according to an 
embodiment of the invention. The window 700 is created by the email messaging 
program, and includes email functionality, such as menu items and toolbar buttons 
702, to, cc, and subject lines 704, and a message text entry area 71 0. The program 
also presents audio or video controls 706 into the window 700, as well as a video 
viewing area 708. Therefore, the user can record video in the same program as he or 
she composes and sends email messages, 

[0049] Conclusion 
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[0050] 



It is noted that, although specific embodiments have been illustrated and 



described herein, it will be appreciated by those of ordinary skill in the art that any 
arrangement is calculated to achieve the same purpose may be substituted for the 
specific embodiments shown. This application is intended to cover any adaptations or 
variations of the present invention. Therefore, it is manifestly intended that this 
invention be limited only by the claims and equivalents thereof. 
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